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The Joint Dissemination Review_Panel IJDRP), part of 
the U.S. Department -of Educations-is one of the most important 
federal mechanisms for recognizing quality in education. Although it 
has reviewed close to 700 projects in its l5-year history, the JDRP 
is still not widely known in the field of evaluation. This paper 
analyzes the JDRP process and its evaluation practices and contrasts 

it with evaluation criter -a of the National Science Teachers 

Associatipa andi^with effective schools: research^ This paper furt^ 
analyzes the JDRP through the use -of _tbe Standards for Evaluations of 
Educational Programs^ Projects and Mater ials^ developed by the Joint 
Coiiiiilittee oil Standards for Educational Evaluation. The standards are 
grouped in four areas^ usability, feasibility, propriety, and 
accuracy. Results highlight those elements of an evaluation to which 
the JDRP attends and how this has influenced the composition of 
projects it reviews and approves. (Author/JAZ) 
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Abstract 



:_ Sespite.having reviewed close to 7^ over 15 year 

history^_tfie U«S« Department of Education's Joint Dissemination 
Review Panel is still not widely known in the field of evalutipn. 
There have also been few stud ies ana lyzitig ^the JDRP prbces s • _ The_ 
purpose of this study was to contrast the JDRP with several other 
recognition processes^ and to analyzetheJDRP. through the use of 
the Standards for Evaluatidns.df Educational Programs , Pro jects 
and Haterials* Results. high light. those elements of an evaluation 
to which the JDRP attends, and how this has influenced the compo- 
sition of projects it reviews and approves. 
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Analyzing tlie JDRP as an Evaluation Process 

by 

-1 Conrad G^Katzenmeyer ^ 

Office of Educational Research and Improvement 
O.S. Department of Education 

and 



Geneva Baertel 
Stanford^ California 



-Recognition of educational quality, whether in students, 
programs, or schools, has always been of interest to the field of 
education, and particularly to the 0*S* Department of Education. 
However, it has received renewed attention with the current 
administrationls concern for excellence. There now exist a number 
of programs charged with the responsibility of identifying 
educational quality. 

Certainly- one of the most important federal Beehahisms for 
recognizing quality in education over the past decade has been 
the Joint Dissemination Review Panel « Givefi= the large number of 
projects that have been reviewed by the JDPf , as well as its role 
with NDN, it is our belief that the JDRP represents one of the 
nostsignificant examples of the use of evaluation in education. 
It is therefore worthwhile to analyze what this process has been 
«nd what it tells us about the practice of educational 
evaluation^ We have chosen to do this by contrasting the JDRP 
process witfi two other approaches that share some of the purposes 
of the JDRP, and to match the JDRP to a set of standards for 
program evaluation that are now generally accepted in the field 
of educational evaluation. 

Alternate Approaches for Recognizing Quality 

Every recognition system is bound to be influenced by the 
nature of the candidate projects it reviews* -For the JD^, a 
major^source of projects until recently was Title ill (Title IVc 
later) of the Elementary and Secondary Education Act. These 
projects had several distinguishing characteristics^ They were 
generally large-scale and well fnnded^ both for develemment and 
evaluation. They also placed major emphasis on the developBeiit of 
materials and prOcessesi thus, they generally generated a 
tangible products Finally, the emphasis was on innovation and 
ehangei The prevailing model was the social science paradigm for 
*®^tj* »ith the accompanying social science approach to 
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The approach to evaluation employed In the JDRP process is the 
one now labeleS "prpgrani evaluation" in the iitlrature* In order 
to meet the requirements for anevaluatibh of this type, the 
program must be a narrow, clearly specified intervention^ |t iust 
be implemented in a standard, verifiable manner i And it must have 
objective, measurable outcomes^ The last of these is critical for 
the JDRPt outcomes are the parts of the program that can be 
addressed host easily and unambiguously by experimental social 
science methodology^ This doesn't necessarily imply aii emphasis 
Oh quantitative methods and standardized achievement tests, btit 
for the most part, this is what hap|)ened with the JDRP* 

However, there are other criteria and techniques that^h^ 
been used for recognizing quality* ftn-interesting contrast is the 
approach that^theHatibhal Science Teachers Association used In 
its Focus on Excellence i The purpose was to identify exemplary 
school science programs ifnl ike projects in some curricular 
areas^ science projects tend to emphasize process variables, and 
seldom rely on standardized achievement tests as euteeme - 
measures! The pro jects were nominated and initially reviewed with 
a process similar^to JDRP's, but site visits were used in the 
final selection. Emphasis was placed on content innovation^ with 
outcomes determined by eenteht specialists* judgments ratJhex than 
tests and quasi-expermimental designs^ This approach is q^ltti 
similar to what HSF used with its curriculum development ^rbjeets 
of the 196DS and early i970s. Despite the differences in 
procedures employed, however, this is also a program evaluatibh 
and the purpose is the same as for the JDRP. 

ipw do these ai^roaches compare? The 55 p^ 
Focus on Excellence in 1982 were retro;^pectively reviewed, using 
the JDRP criteria|Katzehmeyer^l985i) The results indicate that 
if the 50 projects had been submitted to the JDRP, two were 
likely to be passedjohe had in fact been approved}, eight were 
doubtfulf and 40 were very unlikely to receive JKIP approvals In 
other words, of 50 nationally acclaimed science programs, between 
eo and 96% would fail JDRP. 

z But what about this ebmparisoh-from the pppcsite perspective? 
WPuld projects that pass^^^ have made it through Focus on 
Excellence or the NSF review? No direct analysis is possibl^ 
because so few science projects have been passed by JDRP^ and 
several of these were funded by NSF. But our experience in 
evaluating a large numbir Of Title III projects and in - 
participating in the NSF review process lead as-to believe- that 
very few Title III projects would have been considered worthy by 
NSF. NSF's emphasis bh high level science content ^ with great 
attention to accuracy and coniprehehsiveness , was 8iiK>iy beyond 
the capabilities of most Title III projects ^ 

- The difference between the NSF and ^RP approach^ is a Matter 
of-values. NSF ^lued state-of-the-art eehteht; whether students 
ifa?n«^^«cience became the responsibility of the teacher and 
local:^ school in how they used the eohteiiti^RP values student 
oatcomesi whether this is oh state-of-the-art content is the 
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responsibility of the teacher and local school in i^lectina the 
program* While this somewhat overstates the value conflictl it is 

?J*?'^««?^i^K'!K***^®^f®° "tri^ingiy few, examples of projects 
th^t contain both superior content and superior demonstrated 
outcomes in science education. ciuwi»,ttatea 

^Another contrast to the JDRP approach is the identification 
process used^in the effective schools literature. The concept of 
effective^schools began with the observation that students in I 
few schools showed levels of academic achievement, as indicated 
by_^8tandardized tests scores in excess of what was being 
achieved in the majority of schools serving the same kinds of 



Schools with^achievement levels substantially higher than 
?1* based on student characteristics, were 

identified ac effective schools* Researchers then studied these 
inSSfof^SJ ^'f?® f"^? to find the keys to their 

success, Jffective schools were found to be those with a strong 
schoolwide^press toward academic learning, princi|>als who were 
strong^instructional leaders, clearly art iculatei and 
consistently enforced disciplinary and attendance policies, and 
reliance on teacher-centered didactic instruction. 

^ fleariy, the test of the jDfiP in idehtifylhg effective 
programs based on treatment-control group comparisons is 

from that of the effective schools researcher, seekina 
statistical outliers and then inferring the reasonlm their ^ 
success. 

^o^The^JBRP examines educational programs 

"lies «Pon statistical comparisons to control groups 

S l5L™®°^^'.?®f^"P^ complex constellations 

Of programs,^policie8 and practices, and relies upOh statistical 
comparisons to a population of similar schools* 

o^pe two kinds of problems bring different t^es of statistical 
evidence^p^bear, and in different ways. Thosa schools or 
fS^^f*?? "^^^^riterla and those 

identifie.d ^ school effectiveness criteria would probably have 
^JBHf .®?vf expectations in focus and scope would 
aiS^^i^ ^^if divergence. Every single program in an effective 
school might^f ail the JDRP criteria. In another school, several 
programs^adgfit be, judged effective through the JDRP criteria, and 
yet the school night well fail the test of effectiveness, 

it- if noting that a number of early Title III projects 

vere total school innovation efforts, not unlike effective 
le fSJnd^ln^raS!' expected, these projects are not to 
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As the HSfft/KSP and effective schools exampiei indicate, 

in which the recognition task can be 
conceptualized, with_^the results depending on the approach taken, 
alvon^fh^ff gf«|8in kinas^ofprolects at the expensl^Sf I?hils? * 
given their different emphases, this is not surprising. The 
critical issue-is to firsts understand the nature of the project 
to be evaluated,^ and then to deterinlne the most meaningful way of 
quality for that type of project, and for the typi If 
aecision to be made. 

Hatching the JDRP Process to Educational Standards 

^-f^cond^aeans of analyzing the JDRP is through the use of 
the Standards for Evaluations of Educational Programs^ Projects 
I3h^S!®^*;^I* f I^P®^ ^y^^* Joint Coamittee on standards far 
fS''SJ!®3*^^*''f*"-?^§"-^*'^ course, the JBRP was not designed with 
the Standards in mind, but since the Standards represent a 
consensus of what good evaluation practice should be, it is 
reasonable to expect a substantial agreement. The thirty 
standards prepared and tested by the Joint Committee are grouped 
in four general areas: Osabiiity, Feasibility, Propriety* and 
Accuracy, These cover a full range of concerns that miglt be 

^^^^^^Ptlon examples are 
provided wxth each Standard to aid in applying it to an 
individual evaluation. 

^t«?y^-"^*'?E provide ah interesting reference point for the 
JTOP^_ because the JPRP primarily reviews a project^s evalultibh 
and the? makes a judgment from this and the results reported by 
the evaluation* Although the ultimate decision is to approve or 
disai^rove^the project, the JDRP does not review the substance of 
naturl^^* usually has only a limited knowledge of its 

^The JDRP process is perhaps best seen as a meta^evaluatibii that 

expectacions for Information from projects through 

adeguacy and*^cbSpellihgness 
of that documentation and information provided in the oral 

?«f«®®^?*^*°"^ ?^ is also important to note that although the 
JDRP sets program evaluation stsndards for projects, the JDRP 
does not conduct a program evaluation* The meta-evaluation is 

5S5!r°l *'ifff^ ?^?^®®f aecision-making that operates 

under a totally different set of procedures than a projeS* 

i^l p *o TOlF the Standards £6 both the 
expectations of^^ for project evaluation submissions # as 

flJiSf^lsfS g^^^ So^'^fllW'^Ma the JM* process itself . The 
fehl sioiiii to the extent that we could interpret 

the process in light of peogram evaluation Results of 

this anaiysis^are reported in Table 1. In order to keep the 
jnaiysis as si^le as possible, for each Standard we sSnply made 
Sol^^S^Si^S^ «S tfie Sta„p a^3 being emplo or Snt^al 
*n-aolng tbi8# we took into consideration the purpose of the 
i?Sdf?L^Srl !l reflecting the StandSldl llln if tie 

gtandards are stated in more general terms than the Jtm mandate. 
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For example, several of tSestanaards speak to the need for being 
responsive to inultiple aodienc'5s and multiple purposss* It was 
our judgment tfiat the JDRP mandate required a more harrow fociis 
by the project i However, where the mandate did not require the 
approach adopoted, we applied the Standard as directly as 
possible. 

In Column I of Table I i the JDRP's expectations of prbjeets 
are summarixedii The JDRP matches well with the Standards in two 
categories: Dsability and Accuracy. For Usability, only Evaluator 
eredibility and Report Disscsminatlon were given HDs (the 
Guidelines do not address either of tbese)^ while all eleven 
standards in the Accuracy category are addreissed. There is 
considerably less agreement in the other two categories: 
feasibility and Propriety* Hbhe of the Peasibility Standards and 
only two of the Propriety standards^Full and Frank Disclosure and 
Bights of Human Subjects- are addressed, the latter only 
indirectly through the program warranty. 

Thus it is clear the JPRP has .^elected to revie^ only certain 
kinds of standards while ignoring others^ Further, this selection 
Js not part of the mandate^ Feasibility and Propriety could have 
been included; but probably were not because they are less 
outcome-oriented than Accuracy and Usability* tack of emphasis on 
these other concerns^ particularly Feasibility, has led the 5DRP 
to adopt an abstract view of evaluation, as if projects could 
set out tp~create an ideal evaluation without constraints, it 
has also spawned a cottage industry of evaluatbrs brought in by 
projects to write submissions to make it appear this was the 
case. 

Looking next at the JDRP process itself ^as summarised i^ 
Column^ II I we find a strikingly different pattern. Overall, 
Usability is at the same high level, but in both the Feasibility 
and Propriety categories there is also a close match with the 
Standards^ For Accuracy there is considerably less utcbiwhic^ 
can be explained in several ways« Firstj the Accuracy standards 
proved the most difficult to apply ta the JDRP process, because 
isany are stated iii terms that are appropriate only to program 
evaluation. For this reason # we chose not to apply four of the 
Accuracy standards to the JDRP process. Another reaibii for the 
lower emphasis on Accuracy in the JDRP process is the limitations 
that exist in its review. There is no reaibii to believe that the 
^RP could aspire to the same level of accuracy through Several 
hours of reading and haps an hour of oral discussion that 
projects obtain through several years and thousands of dollars of 
evaluation activity* However j this is not Intended to dismiss any 
concerns regarding the-accuracy of the J^ process. It is bur 
belief that there are problems with accura&y, the most glaring 
being the absence of evidence regarding reliability and validitv 
of the decisions made. 
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^ ^ Table I ^ 

Comparison of Standards to the JDRP Expectations 
for Projects and the JDRP Process of Review 



Standards* 
Osability 

Al Audience Identification 

A2 Evaluatdr Gredibility _ 

A3 Information Scope and Selection 

A4 Valuatibnal Interpretation 

A5 Report Clarity^ 

AS Report Dissemination 

A7 Report Timeliness 

AS Evaluation Impact 



FEASIBILITY 

Bl Practical Procedures 
B2 Political Viability 
B3 Cost Effect ivness 



PROPRIETY 

CI formal Obligation 

C2 Conflict of Ihterest- 

C3 Full and Fran§ Disclosure 

e4 Public's Right to Kno?? 

£5 Rights of Human Subjects 

f'6 Human Interactions 

C7 Balanced Reporting 

C8 Fiscal Responsibility 

ACCURACY 

ei Object Identification 

D2 Context Analysis - 

D3 Described Purposes and Procedures 

D4 Defensible lofor&atien Souf ees 

D5 Valid Beasts rement ^ 

Di Reliable Heasurment- - 

D7 Systematic Data Control ^ 

D8 Analysis of Quantitative Information 

D9 Analysis of Qualitative Inforination 

D16 Justified Conclusions 

t>ll Objective Reporting 



I 

Project 
Submission 



YES 

NO 

YES 

YES 

YES 

m 

YES 
YES 



11^ 
JDRP 
Process 



NO 
NO 
NO 



NO 

NO 

YES 

N0 = 

YES 

NO 

NO 

NO 



YES 
YES 
YES 
YES 
YES 
YES 
YES 
YES 
YES 
YES 
l^S 



YES 

YES 

YES 

NO 

NO^ 

YES 

YES 

YES 



YES 
YES 
YES 



YES 
YES 
NO^ 
YES 
YES 
YES 
YES 
YES 



m 

YES 
** 

MO 

NO 
** 

«-* 

•« . 

YES 

YES 



-* Joint Conttitteeeen_stanaard8 for Educat 

Stasaards for -Evaluations- of Educational Programs, Projects and 
"-*-erialSi M Yori; M^rav Hill r 1981. ^ ^ 
EHXC''^ were lisiable to apply this Standard, as written to the JDRP 
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|. The JDRP has heeh an Important part of educational evaluation 
for more than a decade^ both because of the number of projects it 
has^revlewej^ and the gate-keeping function it plays for the 
Kational Diffusion Metworlc. 

l,Jn fulfilling this role, the JPRP has adopted a particular 
point^of view and emphasis in evaluation- the judgment of impact 
as determined through the quasi-riperimental methods of the 
social sciences. Recognitioh systems that have adopted other 
^^il®"^^* »ethdds have achieved different results* While the 
ifDRP process may have served well in judging the quality ©f many 
projects, it would not have served as well in judging sbleEse or 
effective schools projects. j ^ ^ 

3j Analysis of the JDRP through thr educational evaluatlcn 
Standards highlights its strong emphasis on Accuracy, and the 
iack^ of emphasis in the areas of Peasibil ity and Propriety . Lack 
of interest in Feasibility has been particularly difficult for 
the practicing_evaluatOis to accept, as it has made it impossible 
for_ theiD to addrsss the constraints under which they have had to 
workt or to argue that the evaluation conducted may have been the 
best possible under the circumstances. 

1* In comparison tP its expectations of the projects it reviews, 
S^fi,^^^^'^^*'^?® ^^^^ ft notably different set of Btandiirds. The 
JDi^ process places more emphasis on the practical consideratibhs 
of Feasibility and £rbpriety> but does not meet the Accuracy 
standards^expected of its projects. Of particular cbhcerh is the 
• ?ffr^^®-5^ regarding the reliability and 

validity of its decisions. Some of these studies would not have 
been difficult to do, and should have been carried out. 

5. Because of its emphasis on-butc6mes determined in particular 
ways, the JDRP has locked itself into a strategy that emphasizes 
cnly a certain kind of project. As the numbers of this type of 
K?^®?*^*?^^^"'^'**®^®^'^"^ particularly to the demise of Title 

has allowed Itself to become obsolete* its major role 
is now to recertify projects it has previously approved. Any 
JS^ nature and mechanics of recognition that the 

been lost^ * brought to new situations has almost certainly 

f • fefi® ^^^f^^^^^^^^'^^l^nt m providing timely feedback 

to the projects coming before it, and to the HSN, it has failed 
to^provide much information about itself of its deliberations to 
f^-f°t?-'^SS^?^*'®^° education and evaluation. If we are correct 
tbat the JDRP has been a signlficaht element in the practice of 
program evaluation, it has also been one of the best kept 
Bee ret 8. 
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that fall short of the idell il 4^11^ I ? e"»PleB 
the greatest co»plsint evaluatote h«»l"i?fi fhf issp. * 
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